The present invention concerns a device and a method for compensating for color shifts, as arise when employing fiberscopes with color picture cameras.
Optical systems in which an image is transferred to an imaging sensor by optics are in widespread use. Without the employment of endoscopes for imaging, many applications in the field of diagnostics, inspection, quality assurance and research, among others, would be impossible today. On the one hand, refractive optical systems are used here, i.e. systems with a rigid construction within which the image is transmitted onto the sensor through a lens arrangement, similar to an objective of a camera. On the other hand, fiber-optic systems are employed, which consist of a great number of ordered light-guiding fibers combined in a bundle, wherein the light is guided onto a sensor by the multiplicity of fibers.
The present preference for refractive optical systems is founded in their image quality, among other things. Where, in the literal sense, much more “flexible” employment is demanded (small, difficult access), high-quality semi-rigid or pliable endoscopes (fiberscopes) with small working diameters and glass fiber image guides have to be used. When employing such a fiber-optic system of several image guides, typically a single image point or a single intensity value is transmitted for every single image guide used. In the case of justifiable diameters of the entire fiber bundle, no arbitrarily large amount of individual fibers can be used. The reduced resolution caused thereby, or the honeycomb structure recorded by the sensor due to the arrangement of the light fiber, partially inhibit the use of such devices.
For example, the image guides of high-quality fiberscopes consist of a regularly ordered bundle of about 3,000-20,000 individual fibers. By contrast, a typical (low-resolution) conventional motion-picture camera has 640×480, i.e. more than 300,000 image points (VGA), for example, and the image signal transported by means of the fibers usually is observed with such a motion-picture camera.
The individual light guides mostly comprise a cladding, so that spurious structures in the observed image result from the cladding, which structures may be smoothed by low pass filters or adaptively reduced by spectral masking, for example. So as to remove the structures introduced by the honeycomb structure and highly disturbing for assessing an image, there already exist solutions that interpolate, on the basis of at first localized fiber centers, a honeycomb-free image on the basis of the brightness information at the fiber centers. Just like the smoothing of the honeycombed cladding structures or by way of its masking in the Fourier space, however, these methods, although increasing the quality of representation of the captured image, have the disadvantage of failing to achieve an actual increase in resolution of the image.
One problem that is to be solved, in general, is dealt with in German Patent DE 4318140A1. It describes how the centers of the light spots imaged on a higher-resolution sensor by the individual glass fibers can be determined by fitting a suitable function to the brightness distribution generated by an individual optical fiber on the two-dimensional sensor. The patent shows how an association of the allocation of the fibers on the input side of the optical fiber bundle with the position of the light spots caused by the fibers on the sensor is possible on the basis of the adapted fiber coordinates.
For the representation and computerized further processing of the images captured by means of fiberscopic endoscopes, if possible, structure-free representation is desired. Due to the great industrial importance of fiberscopes for the inspection of cavities of technical components as well as for medial diagnostics and therapy of inner organs, or the use as a positioning aid in automation, high-quality true-color representation of the image produced by means of the endoscope is increasingly being demanded.
The previously used algorithms and methods utilize gray scale images or the individual color channels of a multi-channel color image for image rendition, i.e. they only use monochromatic intensity values of an image.
Color cameras, i.e. sensors with several intensity channels (sensor elements that are sensitive to different spectral regions each), cannot be used with the known methods in their present form. In an alternative approach, followed up to now, for using color captures, filtering and/or interpolation is performed for rendering fiber-optic captures of a color camera on the individual color channels of an image (the basic colors red, green and blue, for example). The separately produced images of the individual color channels are then combined so as to attain a multi-channel color image. Regarding color fidelity of the produced images, however, this procedure does not lead to optimum results. If an image that is produced in such a way, for example, is to be used for further visual use, assessment of image data or extraction and processing of color features, this is achieved only to an unsatisfactory extent, owing to the color shifts resulting from a simple combination of a fiberscope with a color camera.
This approach fails because of the false colors caused by the pattern of the sensor elements with different spectral sensitivity and the non-uniform illumination of the sensor elements by an individual optical fiber. For example, arrangement of the pixels sensitive in the red, green and blue spectral regions is customary in the so-called Bayer pattern in CCD sensors and CMOS sensors. This pattern is characterized in that, in a two-dimensional matrix of sensor elements, sensor elements of identical spectral sensitivity do not border on each other in a horizontal or a vertical direction. Pixels sensitive to red, green and blue light are used here, with the number of pixels sensitive to green light being predominant and exactly twice as high as the number of red-sensitive and/or blue-sensitive pixels. As will be explained in greater detail further below, non-uniform illumination of such a color pattern, and image processing electronics of the color camera optimized for uniform illumination, leads to the sensor and/or the color camera producing false-color image information for the sensor elements illuminated by a single optical fiber.
The problem is further aggravated in the digitization of the image by a fiber bundle, because the imaging faces a trade-off. If an individual fiber were to terminate directly and exclusively on an individual sensor element, the intensity could exactly be determined electronically. However, determination of a color value would only be possible for one color component for the corresponding fiber. This means, although it would physically be known which color component it is, no statement can be made concerning the color composition. In order to determine a color value, it thus is desirable to map each fiber onto several sensor elements (to 3×3 or 5×5 sensor elements, for example). However, an undesired color shift results from the imbalance of stimulated sensor elements following the conversion of raw intensity data of the Bayer pattern to color values.
This is due to the fact that the individual pixels of the camera are not illuminated with identical intensity due to the irregular geometry of the optical fibers within the fiberscope. For example, while four red-sensitive pixels are illuminated by a fiber, it is possible that the neighboring fiber only illuminates three red-sensitive pixels. Assuming uniform illumination (as underlying the algorithm operating in a color camera), if a color value and/or several color values are computed from the pixels thus illuminated, this inevitably leads to an undesired color shift. This color shift is perceived as extremely disturbing by the observer and also leads to further undesired effects through abrupt color changes (features in the image).
For example, if motion compensation and/or motion estimation is to be performed due to such a false-color image, this is hardly possible, because the motion estimation is particularly based on finding identical intensity patterns in different image regions of various consecutive captures. If the individual captures per se have false colors, i.e. have corrupted intensities, the motion compensation algorithm cannot achieve any reliable result either.